Color interpolation system and method thereof

ABSTRACT

A color interpolation system is disclosed. An enhancement unit receives raw signals from an image sensor, and then processes the raw signals to output an enhanced first signal. A first interpolation unit receives and processes a raw first signal and accordingly outputs an interpolated first signal. A gain generator generates an enhancement gain according to the enhanced first signal and the interpolated first signal.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to digital image processing, andmore particularly to a color interpolation system and method thereof.

2. Description of Related Art

As an image sensor is commonly equipped with a color filter array (CFA),for example arranged as Bayer pattern, Raw image data retrieved from theimage sensor therefore need be subjected to demosaicing to reconstruct afull color image. The demosaicing is normally performed by interpolationtechnique.

Image enhancement is another digital image processing commonly used toenhance one or more image parameters such as contrast, sharpness, etc.The image enhancement may be performed according to the results of edgedetection or gradient.

In a conventional imaging system, the raw image data retrieved from theimage sensor are firstly subjected to interpolation operation, followedby image enhancement process. One disadvantage of the conventionalimaging system is its inherent drawback that red data and blue data aresampled at a rate substantially lower than green data, such that theimaging system generates distorted image with low sharpness.

For the foregoing reasons, a need has arisen to propose a novel imagingsystem with improved quality.

SUMMARY OF THE INVENTION

In view of the foregoing, it is an object of the embodiment of thepresent invention to provide a color interpolation system and methodthat are capable of simultaneously performing interpolation andenhancement with improved sharpness.

According to one embodiment, the color interpolation system includes anenhancement unit, a first interpolation unit and a gain generator. Theenhancement unit is coupled to receive a number of raw signals from animage sensor, wherein, the raw signals include at least a raw firstsignal and a raw second signal, and the enhancement unit processes theraw signals to output an enhanced first signal. The first interpolationunit is coupled to receive and process the raw first signal andaccordingly output an interpolated first signal. The gain generator isconfigured to generate an enhancement gain according to the enhancedfirst signal and the interpolated first signal.

According to another embodiment, the color interpolation system includesan enhancement unit, a first interpolation unit, a blender and a gaingenerator. The enhancement unit is coupled to receive a number of rawsignals from an image sensor, wherein, the raw signals include at leasta raw first signal and a raw second signal, and the enhancement unitprocesses the raw signals to output an enhanced first signal. The firstinterpolation unit is coupled to receive and process the raw firstsignal and accordingly output a plurality of interpolated first signals.The blender is configured to give weighting to the enhanced first signalwith respect to one of the interpolated first signals, therebyoutputting a weighted enhanced first signal. The gain generator isconfigured to generate an enhancement gain according to the weightedenhanced first signal and the interpolated first signals except theinterpolated first signal fed to the blender.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a block diagram illustrative of a color interpolationsystem according to one embodiment of the present invention;

FIG. 1B shows a block diagram illustrative of a color interpolationsystem according to an embodiment alternative to FIG. 1A;

FIG. 2 shows a process window of image pixels with the raw blue signalat its center;

FIG. 3 shows another process window of image pixels with the raw greensignal at its center, which has raw blue signals to its right and left;

FIG. 4A shows a block diagram illustrative of a color interpolationsystem according to another embodiment of the present invention; and

FIG. 4B shows a block diagram illustrative of a color interpolationsystem according to an embodiment alternative to FIG. 4A.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a block diagram illustrative of a color interpolationsystem 1A according to one embodiment of the present invention. Eachblock of the illustrated system 1A may be implemented by hardware,software or their combination.

In the embodiment, the system 1A is coupled to receive a raw greensignal G_raw, a raw red signal R_raw and a raw blue signal B_raw, whichare in a raw format and are outputted from an image sensor (not shown inthe figure) equipped with a color filter array (CFA) having a patternsuch as Bayer pattern. Other possible patterns may, for example, be RGBpattern or CMY pattern. The color filter array of Bayer pattern may haveplural of color filters, 50% of which are green, 25% of which are redand 25% of which are blue. Generally speaking, there are more greenelements than red elements or blue elements in the Bayer pattern orother similar pattern. In this specification, the green signal, the redsignal and the blue signal may be interchangeably named the firstsignal, the second signal and the third signal respectively.

FIG. 2 shows a process window of image pixels (with the raw blue signalB_raw at its center), which constitute a portion of the image sensor. Anenhancement unit 10 (FIG. 1A) is coupled to receive and process the rawsignals G_raw, R_raw and B_raw, thereby outputting an enhanced greensignal G22_en, which is located, at the center of the process window.The enhancement of the enhancement unit 10 may be performed by anenhancing mask, which may generally be a high-boost filter, as follows:G22_(—)en=W01*P01+W03*P03+W10*P10+W12*P12+W14*P14+W21*P21+W23*P23+W30*P30+W32*P32+W34*P34+W41*P41+W43*P43;

//W01-W43: Enhancing Mask Coefficient.

A first interpolation unit 11 receives and processes the raw greensignal G_raw, and then outputs an interpolated green signal G22_sm atthe center of the process window. A conventional interpolationtechnique, such as bi-linear interpolation, bi-cubic interpolation,spline interpolation, or the like, may be adopted by the firstinterpolation unit 11.

Similarly, a second interpolation unit 12 receives and processes the rawred signal R_raw, and then outputs an interpolated red signal R22_sm atthe center of the process window. A conventional interpolationtechnique, such as bi-linear interpolation, bi-cubic interpolation,spline interpolation, or the like, may be adopted by the secondinterpolation unit 12.

Similarly, a third interpolation unit 13 receives and processes the rawblue signal B_raw, and then outputs an interpolated blue signal B22_smat the center of the process window. A conventional interpolationtechnique, such as bi-linear interpolation, bi-cubic interpolation,spline interpolation, or the like, may be adopted by the thirdinterpolation unit 13.

In one embodiment, a blender 14 may be selectively used to generate aweighted enhanced green signal WG22 according to the enhanced greensignal G22_en and the interpolated, green signal G22_sm. Specifically,the blender 14 is used to give weighting to the enhanced green signalG22_en with respect to the interpolated green signal G22_sm. Theweighted enhanced green signal WG22 may be obtained, for example,according to the following expression:WG22=(G22_(—) en*weight+G22_(—) sm*(1−weight));

//weight: Weight of enhancing mask, not greater than 1 and not less than0.

The expressions demonstrated above are with respect to the processwindow with the raw blue signal B_raw at its center. Similar expressionsmay be likewise obtained with respect to a process window with the rawred signal R_raw at its center.

FIG. 3 shows another process window of image pixels with the raw greensignal G_raw at its center, which has raw blue signals B_raw to itsright and left.

Regarding this process window, the associated enhanced green signalG22_en and the weighted enhanced green signal WG22 may be respectivelyexpressed as follows:G22_(—)en=W00*P00+W02*P02+W04*P04+W11*P11+W13*P13+W20*P20+W22*P22+W24*P24+W31*P31+W33*P33+W40*P40+W42*P42+W44*P44;

//W00-W44: Enhancing Mask CoefficientWG22=(G22_(—) en*weight+G22_(—) sm*(1−weight));

//weight: Weight of Enhancement Mask, not greater than 1 and not lessthan 0.

The expressions demonstrated above are with respect to the processwindow with the raw green signal G_raw at its center, which has raw bluesignals B_raw to its right and left. Similar expressions may be likewiseobtained with respect to a process window with the raw green signalG_raw at its center, which has raw red signals R_raw to its right andleft.

The enhanced green signal G22_en (or the weighted enhanced green signalWG22 if the blender 14 is used) and the interpolated green signal G22_smare fed to a gain generator 16 generates an enhancement gain D accordingto the enhanced green signal G22_en (or the weighted enhanced greensignal WG22) and the interpolated green signal G22_sm. In one example,the enhancement gain D is generated according to a difference betweenthe weighted enhanced green signal WG22 and the interpolated greensignal G22_sm. In another example, the enhancement gain D is generatedaccording to a ratio between the weighted enhanced green signal WG22 andthe interpolated green signal G22_sm.

In one embodiment, if the absolute value of the difference between theweighted enhanced green signal WG22 and the interpolated green signalG22_sm is not greater than a predetermined noise threshold, theenhancement gain D has a value of “1.” Otherwise, the enhancement gain Dis determined according to the value of the difference between theweighted enhanced green signal WG22 and the interpolated green signalG22_sm and may be adjusted by a predetermined value.

Finally, a gain multiplier 17 is configured to multiply theinterpolated, green signal G22_sm, the interpolated red signal R22_smand the interpolated blue signal B22_sm with the enhancement gain Drespectively as follow:G22′=G22_(—) sm*D;R22′=R22_(—) sm*D;B22′=B22_(—) sm*D.

FIG. 1B shows a block diagram illustrative of a color interpolationsystem 1B according to an embodiment alternative to FIG. 1A. The system1B of FIG. 1B is similar to the system 1A of FIG. 1A with thedistinctness being described below. In the present embodiment, only theraw green signal G_raw is processed by an interpolation unit (i.e., thefirst interpolation unit 11). The gain multiplier 17 is configured tomultiply the interpolated green signal G22_sm, the raw red signal R_rawand the raw blue signal B_raw with the enhancement gain D respectively.

FIG. 4A shows a block diagram illustrative of a color interpolationsystem 4A according to another embodiment of the present invention. Eachblock of the illustrated system 4A may be implemented by hardware,software or their combination.

The system 4A of FIG. 4A is similar to the system 1A of FIG. 1A with thedistinctness being described below. In the present embodiment, a firstinterpolation unit 11B receives and processes the raw green signalG_raw, and then outputs interpolated green signals G11-G33 (e.g., 3×3pixels) that constitute a process sub-window (i.e., a subset of theoriginal process window) centered at the process window as shown in FIG.2. For example, the interpolated green signals G11-G33 may be obtainedas follows:G11=f(P01,P10,P21,P12);G12=f(P01,P03,P21,P23,P12);G13=f(P03,P12,P14,P23);G21=f(P10,P12,P30,P32,P21);G22=f(P12,P21,P23,P32);G23=f(P14,P12,P34,P32,P23);G31=f(P21,P30,P41,P32);G32=f(P21,P23,P41,P43,P32);G33=f(P23,P32,P34,P43);G22=G22; //Output Green pixel.

A blender 14B is used to give weighting to the enhanced green signalG22_en with respect to the center interpolated green signal G22 (fromthe first interpolation unit 11B). The weighted enhanced green signalWG22 may be obtained, for example, according to the followingexpression:WG22=(G22_(—) en*alpha+G22*(1−alpha));

//0=<alpha<1.

In the embodiment, the gain generator 16 may perform low-pass filtering(LPF) on the weighted enhanced green signal WG22 and the interpolatedgreen signals, for example G11-G33 except G22, thereby resulting in aninterpolated green signal G22_sm.

Similar to the system 1A of FIG. 1A, the weighted enhanced green signalWG22 and the interpolated green signal G22 are fed to the gain generator16, which generates an enhancement gain D according to a difference or aration between the weighted enhanced green signal WG22 and theinterpolated green signal G22. The gain multiplier 17 is configured tomultiply the interpolated green signal G22_sm, the interpolated redsignal R22_sm and the interpolated blue signal B22_sm with theenhancement gain D respectively.

The expressions demonstrated above are with respect to the processwindow with the raw blue signal B_raw at its center (FIG. 2). Similarexpressions may be likewise obtained with respect to a process windowwith the raw red signal R_raw at its center.

FIG. 4B shows a block diagram illustrative of a color interpolationsystem 4B according to an embodiment alternative to FIG. 4A. The system4B of FIG. 4B is similar to the system 4A of FIG. 4A with thedistinctness being described below. In the present embodiment, only theraw green signal G_raw is processed by an interpolation unit (i.e., thefirst interpolation unit 11B). The gain multiplier 17 is configured tomultiply the interpolated green signal G22_sm, the raw red signal R_rawand the raw blue signal B_raw with the enhancement gain D respectively.

Although specific embodiments have been illustrated and described, itwill be appreciated by those skilled in the art that variousmodifications may be made without departing from the scope of thepresent invention, which is intended to be limited solely by theappended claims.

What is claimed is:
 1. A color interpolation system, comprising: anenhancement unit coupled to receive a plurality of raw signals from animage sensor, wherein the plurality of raw signals include at least araw first signal and a raw second signal, and the enhancement unitprocesses the raw signals to output an enhanced first signal; a firstinterpolation unit coupled to receive and process the raw first signaland accordingly output an interpolated first signal; a gain generatorconfigured to generate an enhancement gain according to the enhancedfirst signal and the interpolated first signal; and a gain multiplierconfigured to multiply the interpolated first signal and the raw secondsignal respectively with the enhancement gain.
 2. The system of claim 1,further comprising a second interpolation unit coupled to receive andprocess the raw second signal and accordingly output an interpolatedsecond signal.
 3. The system of claim 2, wherein the gain multiplier isconfigured to multiply the interpolated first signal and theinterpolated second signal respectively with the enhancement gain. 4.The system of claim 1, wherein the enhancement unit comprises anenhancing mask that acts as a high-boost filter.
 5. The system of claim1, wherein the enhancement unit and the first interpolation unit performon a process window consisted of a plurality of image pixels of theimage sensor.
 6. The system of claim 5, wherein the process window ofimage pixels is configured as a Bayer pattern.
 7. The system of claim 1,further comprising a blender disposed between the enhancement unit andthe gain generator, the blender being configured to give weighting tothe enhanced first signal with respect to the interpolated first signal.8. A color interpolation method, comprising: receiving a plurality ofraw signals from an image sensor, wherein the plurality of raw signalsinclude at least a raw first signal and a raw second signal, which areprocessed to output an enhanced first signal; processing the raw firstsignal to output an interpolated first signal; generating an enhancementgain according to the enhanced first signal and the interpolated firstsignal; and multiplying the interpolated first signal and the raw secondsignal respectively with the enhancement gain.
 9. The method of claim 8,further comprising a step of processing the raw second signal to outputan interpolated second signal.
 10. The method of claim 9, furthercomprising a step of multiplying the interpolated first signal and theinterpolated second signal respectively with the enhancement gain.